The present application relates to image processing and in particular to electronic and computer techniques to enhance the editorial effects that can be produced with motion picture images resulting from film or video.
Machines to produce electrical signals from motion picture film have been known for many years. References exist showing that such processes were known in the 1920""s and used by John Logie Baird. Examples of current machines (xe2x80x98telecine machinesxe2x80x99) to perform this function include the URSA xe2x80x98Diamondxe2x80x99 machine manufactured by Cintel International Ltd. of Ware, Hertfordshire, and the xe2x80x98SPIRIT Datacinexe2x80x99 from Philips, Darmstadt, Germany.
These systems permit some editorial adjustment to the electrical images produced from the film to be made. These functions usually include the adjustment of gain (white level), lift (black level), and gamma (contrast and midtone). Some telecine machines also contain primary and secondary colour correction systems. Primary colour correction is the adjustment of the primary colour content of the image. For example, all areas containing red may be made redder i.e. the level of red in the whole image may be increased. This will necessarily affect the grey tones which are composed of equal components of red, green, and blue. Secondary colour correction is the adjustment of particular colours in the image and, for example, distinctions may be made between areas containing solely red, and areas containing red in conjunction with other colours, such as grey areas.
It is commonly preferred for telecine users and owners to purchase separate colour correction systems to those included in some telecine machines. An example of such a system is the xe2x80x98DCPxe2x80x99 system manufactured by Pandora International Ltd. Many aspects of this system are disclosed in U.S. Pat. No. 5,450,500, International patent application WO 95/12289, and others.
As is disclosed in U.S. Pat. No. 5,450,500, the DCP system is capable of identifying a particular area of an image and, if required, adjusting the appearance of the image to achieve a desired effect. The area to be adjusted may be identified by one or more criteria including the luminance, hue, saturation, position or texture of one or more of its constituent pixels. The DCP is equipped with eight channels each of which can be configured to perform a separate adjustment on an individually defined area of the image. Each pixel in the stream of digital video data from a telecine machine or video source is tested sequentially by each successive channel of the DCP for correspondence with the predetermined criteria stored in that channel. If the pixel meets the criteria of a particular channel it is tagged for adjustment in accordance with the adjustment parameters defined in that channel. The adjustment parameters may include adjustment of one or more of the luminance, hue, saturation or position of the pixel. Thus, each of the channels will adjust the appearance of the pixel under test if it meets the adjustment criteria of that channel. The channels are designed such that each channel in turn can test the input video data stream at a sufficiently high speed that the DCP produces a corrected video data stream at broadcast rate.
There is, however, a trend in the television and video post-production industry to move away from the traditional xe2x80x98clockedxe2x80x99 processing of television images, whereby video frame rates are matched to the broadcast rate of 50 or 60 Hz, so that each television line is processed in exactly one 625th or 525th of the picture frame rate. Modern trends are to process images asynchronously, where data calculations take as long as necessary, provided that the average data processing rate matches the expected rates. Sometimes, it is not even necessary to match average real time or broadcast rates, as some processing in this xe2x80x98computerxe2x80x99 format is acceptable in non-real time.
Traditionally, video data has been stored on video tape and, in particular, digital video data has been stored on digital video tape. However, digital video tape can only be used to record data at real time rates. The advent of large capacity magnetic storage media such a hard disks has meant that video data can be stored and accessed at rates faster than real time. Video colour correctors on the other hand remain configured to process video data at real time rates.
Viewed from one aspect the present invention provides a method for the processing of video data in which the appearance characteristics of picture elements are modified using digital apparatus which provides a plurality of channels having respective processing means, each channel being adapted normally to operate to process a discrete range of characteristics, wherein the method comprises the step of operating the apparatus in an alternative mode in which at least two of the channels are used to carry out identical processing steps, spatial segments of the video data being split between said at least two channels to thereby increase the speed of processing.
Preferably the digital apparatus is a digital colour processing unit in which video data may be passed in sequence through selection means for each channel which determines whether pixels are to be modified in accordance with selection criteria for that channel, and wherein the digital colour processing unit is further provided with routing means for operating the unit in an alternative mode in which data is passed in parallel through at least two of the selection means, said at least two of the selection means having identical selection criteria and the associated channels modifying selected pixels in an identical way.
Viewed from another aspect the present invention provides a method of processing a stream of video data, wherein the data stream is split into a plurality of sequential portions and the portions are routed to a corresponding plurality of processing units each configured to perform an identical operation on the data.
Viewed from another aspect the present invention provides apparatus for the processing of a continuous stream of video data, the apparatus comprising a plurality of processing units for processing the video data and routing means for routing successive portions of the data stream respectively to at least two of the processing units, when said at least two processing units are configured to perform an identical processing operation on the video data.
In accordance with the present invention the video data stream may be processed faster than the individual capability of each processing unit as each processing unit will be operating on only a portion of the video data. The processing units will therefore operate on the video data in parallel when they are performing identical operations.
The number of portions into which the data stream is split will be determined in practice by reference to the number of available processing units and the number of separate simultaneous operations required. For example, in a system of six processing units, or channels, required to carry out only two modifications, the video data stream will be split into three successive portions and each portion will be routed through two processing units, each of the two processing units performing a respective modification.
Reconstituting means may also be provided for reconstructing the split portions of the data stream back into a single stream after processing.
The number of processing channels used may be altered dynamically. For example, an image may require only a simple correction for one part of the image, such as the adjustment of one colour, and this may be done using one processor only. Elsewhere in the image there may be a region where a complicated change is required, involving lots of separate colour adjustments. Additional processing channels may be called in and at least two channels used simultaneously to carry out the same steps. Subsequently these processors may be released.
Adopting computer processing techniques can lead to great productivity increases if intelligent routing is applied to the mapping of algorithms onto the available hardware in a system such as the Pandora DCP. Take as an example the processing of data of a film resolution image in a hardware system such as the DCP. With film resolution work there is no concept of real time, as there is in broadcast television, because the processing operations required are so computationally intensive. It is merely required to process images as cost effectively and efficiently as possible. As disclosed in U.S. Pat. No. 5,450,500, colour correction systems often have six or more dedicated channels for colour correction. Traditionally, it has been common to have each of these channels configured to process one sixth of the colour space or colour hue circle. The colour hue circle can be thought of as a continuous circular field of colours from red to yellow to green to cyan to blue to magenta and back to red. Thus the six conventional channels on a colour corrector are red, yellow, green, cyan, blue, and magenta. According to U.S. Pat. No. 5,450,500 these channels are, however, completely configurable by the operator. Thus, six different areas of the red hue, for example, can be identified, and processed accordingly.
According to the present invention if the operator wishes to make a change only to one red area, all six of the colour processing engines may be configured to process different spatial sixths of the image, to attain processing of the aforementioned film resolution data image in one sixth of the time that would be usual in a conventional synchronous colour processing system. Thus, the data stream corresponding to the image may be routed in six sequential portions to respective channels of the colour processor. Methods for splitting the image between colour processors are many. One possible method is to alternate between processors along the line direction of the image, feeding the first pixel on the line to the first processor, the second pixel to the second processor, and so on, such that successive channels receive successive pixels in a pattern that is repeated after the sixth pixel in the line. Another method is to split the lines between processors, such that all of the first line is processed by the first processor, all of the second line is processed by the second processor, and so on. Alternatives to these methods are many, and include, for example, the processing of the first sixth of the lines of a frame by the first processor, the second sixth by the second processor, and so on. Alternatively, each line may be split into six continuous portions, one portion being routed to each channel.
Similarly, if the operator wishes to make two alterations, to two separate colour regions in the picture, then three times the speed of processing can still be achieved than would have previously been possible. In this case, the six channels will be split into two equal groups each channel in a group performing an identical colour correction operation. Each group will receive all of the picture information in the video data stream, but each channel within the group will only receive one third of the pixels in the data stream for processing.
Other areas of processing to which this technique is applicable include the simple lookup techniques used to alter the gain, lift, and gamma. These functional units can run at many times the speed of the colour channel processor, which in turn can usually run at real time in a video configuration. Conventionally, in the video domain, there would be no point in running adjustments or processing at higher than real time video rate, but with the advent of film resolution data, it is useful to be able to process the data at the maximum rate possible.
Advantageously, some of the channels may be configured in software rather than the hardware configuration such as that of the DCP. Particularly advantageously a mixture of software and hardware channels may be used with the routing taking into account the relative capabilities of each channel. For example, to get maximum throughput of data in a configuration where four channels of colour processing are required, in a hardware/software system that has six channels available of hardware, it may be possible to run at xe2x80x98double speedxe2x80x99 with the hardware, processing colour channel one with hardware channels 1 and 2, colour channel two with hardware channels 3 and 4, colour channel three with hardware channels 5 and 6, whilst processing the fourth colour channel in software. There are many other ways in which a mixed environment of hardware and software processing elements can be matched into a xe2x80x98load balancedxe2x80x99 combination, by the use of intelligent routing.
One hardware implementation of the invention could use a plurality of xe2x80x9cPentiumxe2x80x9d(trademark) or equivalent chips which can operate at speeds of 333 MHz or even more. Alternatively there could be used specialist processors such as the TMS 320C6X of Texas Instruments. This range includes both fixed and floating point chips. A third alternative is the xe2x80x9cNetwork Sharexe2x80x9d range from Analog Devices in the United States. Alternatively, large field programmable gate array (FPGA) elements could be used, such as the 10K150 element from Altera in the United States.